Global Illumination - Solutions

ReSTIR GI

Lumen

  • Lumen - Unreal Engine 5 .

    • We settled on Mesh Signed Distance Fields for our Software Ray Tracing geometry representation. These give reliable occlusion, all areas have coverage, and we still get fast software ray tracing through sphere tracing, which skips through empty space. The intersection with the distance field surface only gives us the hit position and normal, we can’t find the material attributes or the lighting.

    • We tried runtime voxelization and voxel cone tracing, but merging geometry properties into a volume causes lots of leaking, especially in the lower mip maps.

    • We also tried voxel bit bricks, where we stored 1 bit per voxel to mark whether it contains geometry or not. Simple ray marching of bit bricks was surprisingly slow and after adding a proximity map for acceleration, we just decided to drop voxels and arrived at a Global Distance Field

  • Radiance Caching for Real-time Global Illumination - 2021 .

    • I watched the first 10 minutes of the video.

  • Keywords:

    • Downsample incoming radiance.

  • Probes:

    • Octahedral atlas with border.

    • 8x8 per probe, resulting in 64 traces per probe.

    • .

    • .

    • .

Screen Space Radiance Caching

  • UE5 - Instead of tracing for every single pixel on screen, we bundle up our rays and we trace from a much smaller set of pixels.

  • Two-Level Radiance Caching - AMD - GDC 2023 .

    • Instead tradeoff between pathtracing and probes.

    • Less noise with less samples, sounds like a good idea.

  • Enshourded - 2024 .

    • SDF Rays.

    • Spatial Cascaded Cache.

    • Froxel Volumes.

    • Etc.

    • Intense.

    • I don't know if it is Forward+ or Deferred Rendering...

Radiance Cascades (RC)

  • Author's channel .

  • Radiance Cascades: A Novel Approach to Calculating Global Illumination - August 2023 .

  • Radiance Cascades - ExileCon 2023 .

  • Implementation of Radiance Cascades in World Space, purely .

  • Exploring Radiance Cascades; soft-shadows, etc .

  • 2D Global Illumination with Radiance Cascades .

  • 3D Demo .

  • Legit Engine .

    • A rendergraph-based graphical framework for Vulkan, in C++.

    • The author of Radiance Cascades contributed to this engine.

    • Radiance Cascades is not implemented in this engine.

  • They require a lot of compute and memory in 3D space.

  • For 2D and screen-space 3D (like in Path of Exile 2) the amount of data you store is small enough that RC is an efficient solution, but it doesn't scale well for large 3D worlds.

  • Path of Exile 2.

  • Godot Discussion:

    • Juan (against):

      • Despite the hype, radiance cascades actually aren't all that practical, or at least aren't any more practical than competing techniques, in world space mode.

      • Radiance cascades shine in screen space mode (which is why they work excellently for Path of Exile), so they're less an alternative to Lumen/SDFGI and more an alternative to SSGI.

    • fakhraldin (in favor):

      • Suggested Radiance Cascades.

      • This GI solution is not restricted to 2D. Why should a probe based GI be restricted to only 2D in the first place? This doesn't make any sense for every graphics developer who got experience with probe based GI techniques, strange.

      • Radiance Cascades are not restricted to screen space but can be expanded to polygonal hardware ray tracing as well. Alexander clearly confirms this here . And again it is strange to claim the opposite to every experienced graphics developer. In fact there are several released triple A titles, which use probe based GI with polygonal hardware ray tracing instead of screen space.

      • Screen Space is still being used even by triple A studios. Even Software Lumen itself does partially use SSGI for details of close objects among other techniques. Remember, Lumen is composed of several GI techniques. Million dollar game productions do rely on Screen Space. You don't come around using screen space for high frequency details in cross-gen graphics. Signed distance fields and voxel solutions are missing high frequency details.

    • Other people against:

      • We both very explicitly say that yes, radiance cascades can be used in world space. Their limitations just don't make them a better or more practical alternative to SDFGI/HDDAGI. We both very explicitly say that radiance cascades could be a great alternative to SSGI.

      • I would like to see someone create a 3D radiance cascades solution (open-world would be even better). I'm a bit skeptical about its performance since, even in screen space, it's not that impressive compared to other screen space solutions, though it does look better.

      • Juan stated in his thread, open-world is actually radiance cascade's greatest weakness as a world space effect.

      • This is actually what makes radiance cascades (as a screen space effect) really good and why it's used in production in Path of Exile: the cost is the same as other SSGI approaches, but the quality is much higher.

    • Neither SDFGI nor HDDAGI are proven as practical solutions. SDFGI is being replaced by HDDAGI due to its inefficacy, and HDDAGI isn't released yet. Neither have effective parity with production-proven GI techniques and are ultimately experimental.

    • Take Avatar Frontiers of Pandora for example. Their technique is currently seen as the most sophisticated dynamic GI solution for cross-gen and next-gen realtime video games.

    • But it may surprise many people that their solution actually shares a similar ground concept of "radiance cascades". They also use a probe based system. Instead of using radiance cascades, they combine different techniques like world space, screen space for high frequency details and ray tracing per hardware RT or compute shaders RT as a fallback. Similar to radiance cascades they implement additional different layers to achieve a wider spectrum of GI. They capture the world's details by "different grades of detection" so to speak.

    • The solution with "radiance cascades" is way less complicated, more performant and more scalable for hardware. Just like with SDFGI and HDDAGI we already use a probe grid. "Radiance Cascades" is just adding hierarchical probe grids with different resolutions to the existing one. This step increases detail capture and quality tremendously at cheap costs even with ray tracing.

    • We don't even need to make additional probe grids mandatory. It could be optional in the editor and even be offered as an in-game option. The more grid levels you can add, the more quality you can achieve according to your liking and machine. It is highly flexible.

    • From a technical standpoint i really don't see insurmountable objections against this solution, as it doesn't even interfere dramatically with the existing one. Rather it can serve as an additional, supportive and optional layer to the basic probe grid. If you don't want to apply it for world space, than don't do it. There are many another ways.

    • "Radiance Cascades" can be combined with world space and hardware rt or compute shaders rt to achieve similar results to ubisoft's GI solution, if not even better. Many features in godot turned out to be short-lived obsolete code. But i really don't see "Radiance Cascades" as such. It rather could serve as a basis for further development and options, which build upon it.

    • Our resources are limited and it would truly be a missed opportunity not to take advantage of this low-hanging fruit from which a great tree could grow.

    • World space radiance cascades has occlusion challenges (sound familiar?) and memory consumption challenges - just like pretty much all real time GI solutions.

    • However due to the specific nature of these challenges for radiance cascades, any practical implementation of world space radiance cascades will be limited to 3, at most 4, cascades.

    • What if you need GI for an open world map, for example? You might be able to make it work beyond that range, but most potential approaches are challenging to implement, and will likely be expensive to run.

    • For example, one potential approach is a cascade of radiance cascades. Literally running the entire thing multiple times at multiple sizes, and interpolating between them.

    • So, you might say: "well maybe it can just be a higher quality short range alternative to SDFGI/HDDAGI?" You may have some misconceptions about the level of quality of world space radiance cascades, compared to its much more impressive screen space counterpart.

    • Watch this video that Sannikov himself posted . Does the blockiness and crawliness of the lighting look familiar to you? Does the volume representation look familiar to you? The quality tradeoffs of world space radiance cascades are very similar to the quality tradeoffs of SDFGI/HDDAGI. By replacing SDFGI/HDDAGI with world space radiance cascades, you are literally swapping it out with a technique that has practically the same quality, but much more limited range. You're gaining nothing, and losing something.

    • For screen space radiance cascades however, that's a different story. Pretty much everyone agrees it's incredible - probably the best screen space GI the industry has to offer right now, both in terms of performance and quality.

    • Author of Radiance Cascades, commenting on the HDDAGI PR:

      • First, I never pitched 3d RC as the ultimate GI solution. I never even pitched it as a good GI solution. I wouldn't even call it practically viable by my standards, to be honest. I'm just saying that it's a direct improvement (in pretty much all parameters) over anything that has a regular grid (or nested grids) of radiance probes. DDGI for example.

      • Second, the screenspace version of RC is only limited to on-screen occluders and light sources if screenspace raymarching is used. However, screenspace cascades can store worldspace radiance intervals (including offscreen geometry) if you have a way of casting worldspace rays, using either a BVH, an SDM or a voxel raymarcher of some sort. The main limitation of this approach is that it only allows storing radiance on the surface of the depth buffer and you can't use it for example for volumetric lighting.

    • Juan apparently thought of an approach that's even better than HDDAGI, and will eventually work on that instead of continuing HDDAGI.

Dynamic Diffuse Global Illumination (DDGI) / Ray-Traced Irradiance Fields

  • "Dynamic Diffuse Global Illumination with Ray-Traced Irradiance Fields" - Majercik 2019 .

    • Dynamic Diffuse Global Illumination with Ray-Traced Irradiance Fields - Majercik 2019 .

    • Much of the talk is about:

      • Fixing very common light leaks in techniques involving light probes.

        • .

        • .

        • .

      • Better automatic placing of Light Probes.

    • "Up to 256 rays per probe per frame, scaled down in some cases".

    • Limitations :

      • Don't sample on the surface, as it's really unstable.

        • 1mm up and you are ok, 1mm down and you are inside the surface, where everything is black.

        • Use a bias to improve this.

  • "Dynamic Diffuse Global Illumination with Ray-Traced Irradiance Fields" - Majercik 2019 .

  • "Scaling Probe-Based Real-Time Dynamic Global Illumination for Production" - Majercik 2021 .

  • "Improving Probes in Dynamic Diffuse Global Illumination" - Rohacek 2022 .

  • DDGI isn't an alternative to RTXGI, but a way to implement it using light probes.

  • Voxel engine with DDGI .

  • Probes :

    • Uses probe volumes (a set of points in space storing irradiance + visibility).

    • A probe typically holds an estimate of irradiance  or sometimes radiance distribution  directly (often SH coefficients).

      • Comparing to PRT, the DDGI Probes represent the current lighting, not a precomputed transfer.

      • PRT probe = precomputed transfer.

        • “This is how light at this spot responds to any lighting, given the static scene geometry.”

      • DDGI probe = runtime-sampled irradiance.

        • “This is the actual indirect lighting at this spot right now, given whatever is in the scene.”

    • Probes store the incoming light in every single direction for rays passing through their center.

    • It may also store depth/visibility information (to reduce light leakage).

    • Each probe gathers lighting from the environment using ray tracing (usually hardware-accelerated).

    • Lighting between probes is interpolated at runtime.

    • When shading a point on a surface we sample the probes that are nearby, and blend for smooth lighting.

  • Requires a set of probes, each storing low-frequency irradiance (spherical harmonics, octahedral maps, or similar).

  • Probes can be updated progressively with ray tracing, amortizing cost across frames.

  • Usually only practical for low-frequency GI (no fine detail).

  • Bias samples for probes that just had changes in irradiance, like importance sampling, could speed convergence after edits.

Screen Space Indirect Lighting with Visibility Bitmask (VBAO) (SSILVB)

Horizon-Based Indirect Lighting

Multi-Scale Ambient Occlusion (MSAO)

  • Efficient screen-space approach to high-quality multiscale ambient occlusion - 2012

  • "I know HBAO+ is released but this method is far cheaper as you can see on that table in the end of the paper (They tested on GTX 460M! and it is 23ms average). I believe this method will give us far superior visuals than just SSAO."

  • Wicked Engine 2024 :

    • This was my favourite SSAO so far because it handles large areas and small detail alike without any noise or temporal issues. It works by computing the AO in a deinterleaved version of the depth buffer that is contained in a Texture2DArray. It computes the AO in multiple resolutions, then upsamples and combines all of them into a final texture with bilateral blurring.

Screen Space Reflection (SSR)

  • Screen Space Reflections - .

  • Used to capture reflections based on the rendered scene (using the previous frame for instance) by ray-marching in the depth buffer.

  • SSR gives great results but can be very expensive .

  • SSR - Godot .

  • SSR - Wicked Engine 2017 Demo .

    • I implemented screen space reflections around two years ago but never showed it off so here you go.

    • This is the simplest technique that I know of. It is using binary search when raymarching in view space.

Distant Environment Probes / Cube Map IBL

  • IBL with Cube Maps - Filament .

  • Lightmaps, Ambient Color, IBL with CubeMaps, Probe-based Lighting .

  • Used to capture lighting information at “infinity”, where parallax can be ignored. Distant probes typically contain the sky, distant landscape features or buildings, etc.

  • The light is assumed to come from infinitely far away (which means every point on the object's surface uses the same environment map).

  • They are either captured by the engine or acquired from a camera as high dynamic range images (HDRI).

  • Irradiance :

    • The whole environment contributes light to a given point on the object's surface.

  • Radiance :

    • The resulting light bouncing off of the object.

  • Incident lighting must be applied consistently to the diffuse and specular parts of the BRDF.

  • Typically, the environment image is acquired offline in the real world, or generated by the engine either offline or at run time; either way, local or distant probes are used.

  • Obviously the environment image must be acquired somehow and as we'll see below it needs to be pre-processed before it can be used for lighting.

  • .

  • Limitations :

    • Implementing a fully dynamic day/night cycle requires for instance to recompute the distant light probes dynamically.

    • As probes only capture basic color information and direction, very shiny surfaces are not very doable.

    • .

    • Far Cry 3 used probes. In that game you wouldn't find any shiny metallic object, as the probes cannot represent that type of lighting very easily.

    • IBL Cube Maps, on the other hand, could represent metals much better.

    • .

    • Images, in particular cubemaps, are a great way to encode such an “environment light”.

  • Processing Light Probes :

    • We saw previously that the radiance of an IBL is computed by integrating over the surface's hemisphere.

    • Since this would obviously be too expensive to do in real-time, we must first pre-process our light probes to convert them into a format better suited for real-time interactions.

    • The sections below will discuss the techniques used to accelerate the evaluation of light probes:

      • Specular reflectance :

        • pre-filtered importance sampling and split-sum approximation

      • Diffuse reflectance :

        • irradiance map and spherical harmonics

  • Implementation (Filament) : 

    vec3 irradianceSH(vec3 n) {
    // uniform vec3 sphericalHarmonics[9]
    // We can use only the first 2 bands for better performance
    return
          sphericalHarmonics[0]
        + sphericalHarmonics[1] * (n.y)
        + sphericalHarmonics[2] * (n.z)
        + sphericalHarmonics[3] * (n.x)
        + sphericalHarmonics[4] * (n.y * n.x)
        + sphericalHarmonics[5] * (n.y * n.z)
        + sphericalHarmonics[6] * (3.0 * n.z * n.z - 1.0)
        + sphericalHarmonics[7] * (n.z * n.x)
        + sphericalHarmonics[8] * (n.x * n.x - n.y * n.y);
}

// NOTE: this is the DFG LUT implementation of the function above
vec2 prefilteredDFG_LUT(float coord, float NoV) {
    // coord = sqrt(roughness), which is the mapping used by the
    // IBL prefiltering code when computing the mipmaps
    return textureLod(dfgLut, vec2(NoV, coord), 0.0).rg;
}

vec3 evaluateSpecularIBL(vec3 r, float perceptualRoughness) {
    // This assumes a 256x256 cubemap, with 9 mip levels
    float lod = 8.0 * perceptualRoughness;
    // decodeEnvironmentMap() either decodes RGBM or is a no-op if the
    // cubemap is stored in a float texture
    return decodeEnvironmentMap(textureCubeLodEXT(environmentMap, r, lod));
}

vec3 evaluateIBL(vec3 n, vec3 v, vec3 diffuseColor, vec3 f0, vec3 f90, float perceptualRoughness) {
    float NoV = max(dot(n, v), 0.0);
    vec3 r = reflect(-v, n);

    // Specular indirect
    vec3 indirectSpecular = evaluateSpecularIBL(r, perceptualRoughness);
    vec2 env = prefilteredDFG_LUT(perceptualRoughness, NoV);
    vec3 specularColor = f0 * env.x + f90 * env.y;

    // Diffuse indirect
    // We multiply by the Lambertian BRDF to compute radiance from irradiance
    // With the Disney BRDF we would have to remove the Fresnel term that
    // depends on NoL (it would be rolled into the SH). The Lambertian BRDF
    // can be baked directly in the SH to save a multiplication here
    vec3 indirectDiffuse = max(irradianceSH(n), 0.0) * Fd_Lambert();


    // Indirect contribution
    return diffuseColor * indirectDiffuse + indirectSpecular * specularColor;
}

  • Implementation (Vulkan-glTF-PBR) :

    • The snippets below come from material_pbr.frag 

    // Calculation of the lighting contribution from an optional Image Based Light source.
// Precomputed Environment Maps are required uniform inputs and are computed as outlined in [1].
// See our README.md on Environment Maps [3] for additional discussion.
vec3 getIBLContribution(PBRInfo pbrInputs, vec3 n, vec3 reflection)
{
    float lod = (pbrInputs.perceptualRoughness * uboParams.prefilteredCubeMipLevels);
    // retrieve a scale and bias to F0. See [1], Figure 3
    vec3 brdf = (texture(samplerBRDFLUT, vec2(pbrInputs.NdotV, 1.0 - pbrInputs.perceptualRoughness))).rgb;
    vec3 diffuseLight = SRGBtoLINEAR(tonemap(texture(samplerIrradiance, n))).rgb;
    vec3 specularLight = SRGBtoLINEAR(tonemap(textureLod(prefilteredMap, reflection, lod))).rgb;
    vec3 diffuse = diffuseLight * pbrInputs.diffuseColor;
    vec3 specular = specularLight * (pbrInputs.specularColor * brdf.x + brdf.y);
    // For presentation, this allows us to disable IBL terms
    // For presentation, this allows us to disable IBL terms
    diffuse *= uboParams.scaleIBLAmbient;
    specular *= uboParams.scaleIBLAmbient;
    return diffuse + specular;
}

    • Code where the IBL is calculated and used. 

    // Bla bla bla, basic lighting.
vec3 F = specularReflection(pbrInputs);
float G = geometricOcclusion(pbrInputs);
float D = microfacetDistribution(pbrInputs);
const vec3 u_LightColor = vec3(1.0);
vec3 diffuseContrib = (1.0 - F) * diffuse(pbrInputs);
vec3 specContrib = F * G * D / (4.0 * NdotL * NdotV);
vec3 color = NdotL * u_LightColor * (diffuseContrib + specContrib);

// Calculate lighting contribution from image based lighting source (IBL)
color += getIBLContribution(pbrInputs, n, reflection);
const float u_OcclusionStrength = 1.0f;

// -> Ambient Occlusion
// Apply optional PBR terms for additional (optional) shading
if (material.occlusionTextureSet > -1) {
    float ao = texture(aoMap, (material.occlusionTextureSet == 0 ? inUV0 : inUV1)).r;
    color = mix(color, color * ao, u_OcclusionStrength);
}

// Emissive
vec3 emissive = material.emissiveFactor.rgb * material.emissiveStrength;
if (material.emissiveTextureSet > -1) {
    emissive *= SRGBtoLINEAR(texture(emissiveMap, material.emissiveTextureSet == 0 ? inUV0 : inUV1)).rgb;
};
color += emissive;
outColor = vec4(color, baseColor.a);

Reflection Probes / Cube Map Reflections

  • Store environment radiance  (not just irradiance ).

  • Stores a cube map texture of the surroundings, representing light incoming from all directions.

  • Usage :

    • Used for specular reflections (environment mapping).

    • Mipmapped cube maps can be used with roughness filtering (for PBR specular).

  • Good fits :

    • Games with a lot of metals, as that's where cube maps "shine" (literally).

    • Racing games are a good fit.

  • Limitations :

    • They are static; if the scene changes, the cubemap needs to be regenerated, and that's expensive.

    • Doesn't handle self-reflections.

      • .

  • Reflection Probes - Godot .

    • Reflection probes are used as a source of reflected and ambient light for objects inside their area of influence.

    • They can be used to provide more accurate reflections than VoxelGI and SDFGI while being fairly cheap on system resources.

    • Since reflection probes can also store ambient light, they can be used as a low-end alternative to VoxelGI and SDFGI when baked lightmaps aren't viable (e.g. in procedurally generated levels).

    • Good reflections, but poor indirect lighting.

    • Indirect lighting can be disabled, set to a constant color spread throughout the probe, or automatically read from the probe's environment (and applied as a cubemap). This essentially acts as local ambient lighting. Reflections and indirect lighting are blended with other nearby probes.

    •   .

    • It interacts with LightmapGI:

      • .

    • Reflection probes can also be used at the same time as SSR to provide reflections for off-screen objects.

      • Godot will blend together the SSRs and reflections from reflection probes.

      • This way you can get the best of both worlds: high-quality reflections for general room structure (that remain present when off-screen), while also having real-time reflections for small details.

      • .

    • To get reasonably accurate reflections, you should generally have one ReflectionProbe node per room (sometimes more for large rooms).

    • The extents don't have to be square, and you can even rotate the ReflectionProbe node to fit rooms that aren't aligned with the X/Z grid.

      • Use this to your advantage to better cover rooms without having to place too many ReflectionProbe nodes.

    • Properties .

    • Blending :

      • To make transitions between reflection sources smoother, Godot supports automatic probe blending:

        • Up to 4 ReflectionProbes can be blended together at a given location. A ReflectionProbe will also fade out smoothly back to environment lighting when it isn't touching any other ReflectionProbe node.

        • SDFGI and VoxelGI will blend in smoothly with ReflectionProbes if used. This allows placing ReflectionProbes strategically to get more accurate (or fully real-time) reflections where needed, while still having rough reflections available in the VoxelGI or SDFGI's area of influence.

      • To make several ReflectionProbes blend with each other, you need to have part of each ReflectionProbe overlap each other's area. The extents should only overlap as little as possible with other reflection probes to improve rendering performance (typically a few units in 3D space).

    • Performance :

      • ReflectionProbes with their update mode set to Always  are much more expensive than probes with their update mode set to Once  (the default). Suited for integrated graphics when using the Once update mode.

  • Rotate the camera 6 times to render the 6 faces of the cube.

  • You only render the plane, not the Tea Pot.

  •   .

  • Instead of using the environment map loaded, you use the cubemap generated. So the reflection ON the Tea Pot only comes from this Cube Map.

  • You should also use the Cube Map on the mirror image of the Tea Pot (made when using Planar Reflections).

  • .

    • The mirror image is clearer now.

  • .

Planar Reflections / Flat Mirror

  • Used to capture reflections by rendering the scene mirrored by a plane. This technique works only for flat surfaces such as building floors, roads and water.

  • Instead of reflecting the entire world, we reflect the camera; it's the same thing.

  • .

  • Wicked Engine 2024 .

    • I’m disappointed whenever I see a modern game not supporting proper mirror reflections or if it has screen space reflection (SSR) on flat water. That’s why in Wicked Engine I’d like to show that planar reflection is still relevant and it should be one of the first choices of a game when a reflection needs to be rendered. The planar reflection is the perfect solution for a mirror because that’s what it was made for, and it’s good enough to use even on a large water surface with waves, like an ocean or a lake. Even though the waves are not totally accurate to be represented, it’s still a lot better than noisy SSR that cuts of abruptly.

    • .

    • In Wicked Engine, the planar reflections are using a second full depth prepass + color pass with all the forward rendering pipeline capabilities. Although most of the secondary effects are turned off for them, simply by not running those passes for planar reflections. Also they don’t generate visibility buffer, only depth buffer in the prepass. Planar reflections rendering is also scheduled in the frame asynchronously to the main camera’s compute effects, so there is also room to utilize the modern graphics API to render them. Compared to the main camera, planar reflections are rendered in quarter of the main camera resolution in both axes so they become less dependent on the pixel shader performance, but more geometry heavy, which helps a bit with async compute passes at the same time. To combat the low resolution look, I choose to render them at 4x MSAA right now for some additional anti-aliasing. Quarter resolution means that the resolution is 1/16 compared to the main camera, and adding 4x MSAA on top doesn’t bring back the full detail, but I found it quite nice, for now, although it can be tweaked easily if needed.